The Importance of Proper Weighting Methods

The impor tance of good weighting methods in information retrieval methods tha t stress the most useful features of a document or query representat ive is examined. Evidence is presented tha t good weighting methods are more impor tan t than the feature selection process and it is suggested tha t the two need to go handin-hand in order to be effective. The paper concludes with a me thod for learning a good weight for a t e rm based upon the characterist ics of tha t term. 1. I N T R O D U C T I O N Other than experimental results, the first part of this paper contains little new material. Instead, it's an attempt to demonstrate the relative importance and difficulties involved in the common information retrieval task of forming documents and query representatives and weighting features. This is the sort of thing that tends to get passed by word of mouth if at all, and never gets published. However, there is a tremendous revival of interest in information retrieval; thus this a t tempt to help all those new people just starting in experimental information retrieval. A common approach in many areas of natural language processing is to 1. Find "features" of a natural language excerpt 2. Determine the relative importance of those features within the excerpt 3. Submit the weighted features to some taskappropriate decision procedure This presentation focuses on the second sub:task above: the process of weighting features of a natural language representation. Features here could be things like single word occurrences, phrase occurrences, other relationships between words, occurrence of a word in a title, part-of-speech of a word, automatically or manually assigned categories of a document, citations of a document, and so on. The particular overall task addressed here is that of information retrieval finding textual documents (from a large set of documents) that are relevant to a user's information need. Weighting features is something that many information retrieval systems seem to regard as being of minor importance as compared fo finding the features in the first place; but the experiments described here suggest that weighting is considerably more important than additional feature selection. This is not an argument that feature selection is unimportant, but that development of feature selection and methods of weighting those features need to proceed hand-in-hand if there is to be hope of improving performance. There have been many papers (and innumerable unpublished negative result experiments) where authors have devoted tremendous resources and intellectual insights into finding good features to help represent a document, but then weighted those features in a haphazard fashion and ended up with little or no improvement. This makes it extremely difficult for a reader to judge the worthiness of a feature approach, especially since the weighting methods are very often not described in detail. Long term, the best weighting methods will obviously be those that can adapt weights as more information becomes available. Unfortunately, in information retrieval it is very difficult to learn anything useful from one query that will be applicable to the next. In the routing or relevance feedback environments, weights can be learned for a query and then applied to that same query. But in general there is not enough overlap in vocabulary (and uses of vocabulary) between queries to learn much about the usefulness of particular words. The second half of this paper discusses an approach that learns the important characteristics of a good term. Those characteristics can then be used to properly weight all terms. Several sets of experiments are described, with each set using different types of information to determine the weights of features. All experiments were done with the SMART information retrieval system, most using the T R E C / T I P S T E R collections of documents, queries, and relevance judgements. Each run is evaluated using the " l l -point recall-precision average" evaluation method that was standard at the TREC 1 conference.